Understanding and manipulating face images in-the-wild is of great interest to the computer vision and graphics community, and as a result, has been extensively studied in previous work. Example techniques range from relighting portraits (e.g., Y. Wang, Z. Liu, G. Hua, Z. Wen, Z. Zhang, and D. Samaras, Face Re-lighting from a Single Image Under Harsh Lighting conditions, Pages 1-8, June 2007), editing or exaggerating expressions (e.g., F. Yang, J. Wang, E. Shechtman, L. Bourdev, and D. Metaxas, Expression Flow for 3d-aware Face Component Transfer, ACM Transactions on Graphics (TOG), volume 30, page 60. ACM, 2011), and even driving facial performances (e.g., F. Yang, J. Wang, E. Shechtman, L. Bourdev, and D. Metaxas, Expression Flow for 3d-aware Face Component Transfer, ACM Transactions on Graphics (TOG), volume 30, page 60. ACM, 2011). Many of these methods start by explicitly reconstructing facial attributes such as geometry, texture, and illumination, and then editing these attributes in the image. However, reconstructing these attributes is a challenging and often ill-posed task. Previous techniques attempt to address these challenges by either utilizing more data (e.g., RGBD video streams) or imposing a strong prior on the reconstruction that is adapted to the particular editing task that is to be solved (e.g., utilizing low dimensional geometry). As a result, these techniques tend to be both costly (with respect to use of computational resources) and error-prone. Moreover, such techniques fail to generalize at scale.